Regarding image capturing devices such as digital cameras or video cameras, their image sensors usually include a red (R), green (G), blue (B) three-color optical filter. A light incident on a camera is separated by such a three-color optical filter, and an image sensor converts the separated lights into image signals, thereby generating RGB image data.
When an image sensor which is used in an image capturing device is a silicon-based sensor, the image sensor is sensitive to a light in the visible light region to the near infrared region. On the other hand, since highly precise color reproduction is required for a digital camera or a video camera, a near infrared light which will adversely affect color reproduction is removed by a near infrared cut filter. This is because, while an R, G, B three-color optical filter secures the transmittance in a wavelength band in the visible light region for which each filter is responsible, there are some cases in which the transmittance characteristics of a light in the near infrared region which is outside the visible light region are not taken into consideration.
FIG. 1 is an example illustrating spectral transmittances of an RGB three-color optical filter. Assuming the visible light region to be wavelengths of 400 nm to 700 nm, each color filter is expected to have characteristics that a light having a wavelength of about 400 nm to 500 nm (a B filter), about 500 nm to 600 nm (a G filter), or about 600 nm to 700 nm (an R filter) is passed through. As illustrated in FIG. 1, it is however confirmed that each filter has characteristics that a light having a wavelength of 700 nm or longer which is outside the visible light region or a near infrared light is passed therethrough.
Incidentally, the spectral sensitivity characteristics of a photodiode image sensor which is generally employed in a color image input device such as a digital camera or a video camera also has a sensitivity in a wavelength region of 700 nm or longer. Only applying a three-color optical filter having the spectral characteristics of FIG. 1 as it is to an image sensor will cause a problem, from the viewpoint of color reproduction.
The color-matching functions of the XYZ color system for color perception of human beings are as illustrated in FIG. 2. Regarding color perception of human beings, the sensitivity to a light having a wavelength of 700 nm or longer is zero, and therefore, a light having a power in a wavelength region of 700 nm or longer does not influence perceived color which is a psychophysical quantity.
Now, as illustrated in FIG. 3, a case of observation of a light having a power in a wavelength region of 600 nm or longer will be taken into consideration. In the case of human beings, the light is perceived as red. On the other hand, when the light is observed with an image sensor by using a three-color optical filter having characteristics illustrated in FIG. 1, output signals of the sensor include not only an R value but also G and B values. As a result, a color which is different from a red color perceived by human beings is observed. As describe above, a cause of a problem of color reproduction is that the spectral transmittances are not zero in wavelength regions of 700 nm or longer of G and B optical filters.
In order to attain a highly precise color reproduction for color perception of human beings in a color image input device, an infrared light (IR: infrared) cut filter having a spectral transmittance which eliminates the influence of a near infrared light having a wavelength of 700 nm or longer as illustrated in FIG. 4 is used. Specifically, as illustrated in FIG. 5, an IR cut filter is incorporated into an optical system of a color image input device to intercept penetration of a near infrared light into a three-color optical filter and an image sensor. Due to such a configuration, a light having a power only in the wavelength region of a visible light is input to the three-color optical filter, and the lights separated by the three-color optical filter are input to the image sensor, whereby RGB signals are generated.
On the other hand, when an image is taken at nighttime out of doors or in a dark place, highly sensitive image capturing with reduced noise is demanded. In such a case, in order to reduce sensor noise caused by an insufficient quantity of light, it is desired that as large a quantity of light as possible is received by an image sensor. In order to attain highly sensitive image capturing in a dark place, a method of picking up an image by utilizing a light having a wavelength in the near infrared region is proposed. Examples of the simplest method include a method in which an IR cut filter set in an optical system is mechanically removed during highly sensitive image capturing. This method, however, not only increases the cost of products due to the increased number of parts thereof but has a major risk that it is highly probable that the product malfunctions during its long period of use since a mechanical action of removing the IR cut filter is required.
As a method of picking up an RGB image and an IR (NIR: Near Infra-Red) image without requiring a mechanical action, a method which uses two cameras capturing an RGB image and an IR image, respectively is proposed in NPL 1.
NPL 2 proposes an image sensor into which a four-color optical filter in which an IR filter passing through a near infrared (NIR) light is added to an RGB three-color optical filter is incorporated, as illustrated in FIG. 6. A second figure of NPL 2 illustrates the spectral sensitivity characteristics of each of the optical filters R, G, B, and IR. The spectral sensitivity characteristics of each of the color filters R, G, and B has a spectral sensitivity similar to that of the IR filter with respect to a near infrared light. When an image is taken in daylight, in order to attain a high color reproduction, influences of near infrared lights involved in R, G, and B need to be eliminated. An image sensor of NPL 2 generates R, G, and B signals while eliminating influences of near infrared lights involved in R, G, and B by utilizing an IR signal obtained by passing through the IR filter. When an image is taken at nighttime, all of the R, G, B, and IR signals are utilized.
PTL 1 proposes an image capturing device which generates R, G, B, and NIR signals by using an R, G, B three-color optical filter through which a near infrared light passes and by using a special photo sensor which senses a near infrared light (NIR). For example, a light which has passed through an R filter corresponds to R+NIR and are made incident to the photo sensor. The photo sensor is composed of a visible light sensor unit which detects R at a shallow position in the light incident direction and a non-visible light sensor unit which detects NIR at a deep position in the light incident direction. Regarding G and B, a similar configuration as mentioned above is employed.
NPL 3 illustrates one example of a method of demosaicing processing, which will be mentioned for explaining the present exemplary embodiment. NPL 4 illustrates a method using Gradient Based Interpolation which will be mentioned below. NPL 5 illustrates one example of a method of demosaicing processing. PTL 2 discloses an imaging device that can remove the influences of unnecessary wavelength region components, such as infrared light, without using an infrared light filter.